Associating first and second watermarks with audio or video content

ABSTRACT

The Present invention relates generally to processing audio or video content. In one embodiment, a method is provided including receiving audio or video content, the audio or video content includes a first watermark associated therewith, the first watermark has first data indicating that the audio or video content is rights protected; and associated a second watermark with the audio or video content, the second watermark including second data indicating permissible uses or restrictions associated with the audio or video content, wherein the second watermark is associated with the audio or video content later in time relative to the first watermark. Other embodiments are described and claimed as well.

This application is a continuation of U.S. patent application Ser. No.09/801,515, filed Mar. 7, 2001 (issuing as U.S. Pat. No. 7,127,744),which claims the benefit of U.S. Provisional Patent Application No.60/188,462, filed Mar. 10, 2000. These patent documents are herebyincorporated herein by reference.

TECHNICAL FIELD

This invention relates to the field of embedded data, known aswatermarks, and copy control for media.

BACKGROUND OF THE INVENTION

Digital content, including audio, video, images, multimedia, etc., iseasy to copy and expensive to create. Thus, it is a great target forillegal distribution, defined as piracy. Currently, this piracy isoccurring with audio, using MPEG-1 Layer 3 (MP3) bit-rate compressionformat and the Internet. The MP3 format is used by new artist who wanttheir music freely distributed, as well as by people transforming CDaudio into MP3 and possibly illegally distributing it on the Internet.Professional artists and record labels want to stop the second actionwhile allowing the first, and even distribute new content in MP3.

The problem with robust embedded data based methods of protectingcontent in this insecure format is that they are computationallyintense. The prior-art describes numerous examples of such methods usingembedded data (a.k.a. watermarks or steganography) that requirefrequency transformations. Assuming a different watermark is requiredfor each user, player, storage unit and/or content, distributors willneed expensive equipment to protect the data and users will requireexpensive devices to render the content.

The problem with efficient embedded data methods used to protect thecontent in this insecure format is that they are easy to remove, eventhough they can be made robust to duplication, such as Utility patentapplication Ser. No. 09/404,291 (now U.S. Pat. No. 7,055,034) entitled“Method and apparatus for robust embedded data” submitted on Sep. 23,1999 by Levy, included herein by reference. Removal of embedded data isnot a problem if you require the content to contain the embedded data inorder to be rendered, but this concept only works in a secure format. Inother words, in an insecure format, if the embedded data that informsthe rendering device that the content can or cannot be played isremoved, the content can always be played. However, with a secureformat, removal of the embedded data that informs the rendering devicethat the content can or cannot be played leaves the content uselesssince the device cannot render, such as decrypt, the content withoutthis embedded authorization. For example, in MP3, an insecure formatwhere there is a desire to freely distribute content without protection,the removal of the watermark creates useful pirated content.

Cryptology can also be used to secure the content. However, not only isthis technique computationally intense since it requires many operationsusing a large number of bits, but also one can argue that the format hasbeen changed since existing players cannot play the protected content.

SUMMARY OF THE INVENTION

This object of this novel process is to efficiently protect content inan insecure format using two different layers of embedding data(referred to as watermarks for ease of understanding).

One watermark is robust and declares that the content is protected. Thiswatermark is embedded when the content is created in the desired format,such as MP3, CD or DVD. This means that the computational intensity ofadding the watermark is not an issue because the watermark is only addedto the audio once, and copied with the audio by the distributor. Thiswatermark is labeled the Protect watermark.

The other watermark gives the content its rights, i.e. declares that itis okay to play or record the content. It is efficient, and does notneed to be difficult to removal, since removing it produces noadvantageous results. The efficiency of this watermark is desirablesince it must be embedded each time the audio reproduced, such asdownloaded on the Internet, possibly to link the content to the user,player, recorder and/or storage device. Thus, it greatly reduces thecost of copy management for the distributor. In addition, it lowers thecost of the portable players, since they only have to find thisefficient watermark. This watermark is labeled the Rights Watermark.

To this end, it is desirable to use different types of watermark foreach layer and not two different layers of one watermarking technique.For example, it is not desirable to use one watermarking technique whereone layer is embedded at a low-level, thus being fragile, and anotherlayer is embedded at a higher-level, thus being robust.

Importantly, non-protected content may contain neither watermark and canbe rendered by any device from any storage. Thus, the rendering devicescan be both forward and backwards compatible.

The invented apparatus, which implements the described process, includesan analog or digital logic processor and a storage unit, such a randomaccess memory.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an overview of the process of using two watermarks to protectcontent in an insecure format with minimal increase in computationalcomplexity, and thus cost.

FIG. 2 displays the pseudocode for the embedding process.

FIG. 3 displays the pseudocode for the retrieving process.

FIG. 4 displays the apparatus.

DETAILED DESCRIPTION

This invention begins by explaining the terminology. Content refers tothe data, including but not limited to audio, video, images and smells.Storage refers to device that stores the data. The term watermark refersto any system of embedding data that is minimally perceived when thecontent is played, and is also known as steganography. Data embedded inthe header, and not hidden within the content is still considered awatermark. Robust methods are difficult to bypass. A pirate is anindividual who attempts to illegally copy or distribute the content.

FIG. 1 displays an overview of the invented process. Content 100 existsin an insecure format, which means that non-compliant devices, i.e.devices ignoring copy protection rules, can render the content even ifthe content declares itself as non-renderable. An intrinsic benefit ofan insecure format is that legacy devices, i.e. device created beforethe copy protection system was defined, can render the protectedcontent. In other words, the system is backwards and forwardscompatible. An example of an insecure format is MP3. Some artists wishto freely distribute their content in this format. However, there areother interested parties who want to distribute their content in thesame format without allowing it to be freely copied and redistributed.

The protect watermark 110 declares that the content is protected. Theprotect watermark 110 must be extremely difficult to remove, and,accordingly, may be computationally intense. Many existing watermarkmethods meet this description, and future ones will certainly bedesigned. The rights watermark 120 gives the user rights to render thecontent. This watermark may link the content to the user, player,recorder and/or storage device. This link would determine if the usermay copy and/or play the content. The rights watermark 120 must be acomputationally efficient method that is hard to duplicate. Currently,Levy's application, as referenced above, describes how to designembedded data that is hard to duplicate, i.e. transfer between contentto give rights to content that should not include these rights. However,it is expected that more duplication resistant watermarks will beproduced in the future.

Both watermarks are embedded and retrieved at different times in thereproduction process, as shown in FIGS. 1, 2 and 3. The protectwatermark 110 is embedded when the audio is created, and copied with theaudio when distributed. In addition, the protect watermark 110 is onlyretrieved when the rights watermark 120 does not exist in the content.Thus, the computational intensity of adding the watermark is not thatimportant.

The rights watermark 120 is embedded when the content is reproduced,such as being distributed, placed on permanent storage, or encoded to analternative form by a personal encoding device. The term reproducedrefers to the legal transformation or distribution of the content,whereas copying refers to an individual producing an exact bit-for-bitreplication of the content for legal or illegal utilization. Sincerights watermark 120 is embedded every time the content is reproduced,its efficiency creates a useful reduction in cost for the supportinghardware. Since the rights watermark 120 is embedded after watermark 110it must be okay to layer the watermarks, as known to be possible withexisting technology.

Optimally, the watermarks are search and retrieved in a specific order,as shown in FIGS. 1 and 3. First, the content is searched for rightswatermark 120 (box 300). If rights watermark 120 is retrieved (box 310)the embedded information is evaluated (box 320). If the embeddedinformation is correct, the desired action is enabled (box 330).Alternatively, if the embedded information is not correct, the desiredaction is disabled (box 340). Only if rights watermark 120 is not founddoes the content need to be searched for the computationally intenseprotect watermark 110 (box 350). If protect watermark 110 declares thecontent protected, then the desired action is disabled (box 340),otherwise the desire action is allowed (box 330).

When using a rendering device, such as a MP3 player, which has aportable section, the watermark processing tasks can be split betweenthe loader, potentially a PC program, and the portable section. Thesplit can be designed such that the portable section never needs toretrieve the protect watermark, thus reducing the price of the consumerelectronics portable player by reducing required processing power inthis portable section. For example, when loading the content to theportable section, the loader can check for the rights watermark and theprotect watermark, if necessary. If the desired action for the contentis not allowed, the content is not loaded. If the desired action isallowed, the content is loaded to the portable device.

Then, the portable device may only required to process the rightswatermark, which is efficient to retrieve and embed, for future actions.The portable section would check for the rights watermark 120 if therights watermark 120 contained information the portable device isrequired to understand, such that the portable device can intelligently(i.e. based upon an rules engine) decide how to act upon the content.For example, Utility patent application Ser. No. 09/522,312 (now U.S.Pat. No. 6,868,497) entitled “Method and apparatus for automatic IDmanagement” submitted on Mar. 9, 2000 by Levy (included herein byreference), requires that the portable section (i.e. portable player)requires the user ID contained in the rights watermark such that theportable section can track usage and intelligently limit it to aspecified number of users, while allowing all content to be previewed.

Finally, this invented process can be used to restrict copying and/orplaying of the content. Since this content is easily created byindividuals and desired to exist on storage in general purposecomputers, it is preferred to use the invention to restrict playing.

FIG. 4 shows the hardware apparatus required to implement the inventedprocesses, such as embedding and detecting the protect watermark 110 andrights watermark 120. The hardware includes a logic processor 400 and astorage unit 410. The logic processor 400 may be defined as theequivalent of a digital signal processor (DSP), general-purpose centralprocessing unit (CPU), or a specialized CPU, including media processors.A likely DSP chip is one of the Texas Instruments TMS320 product line. ACPU could include one of Intel's Pentium line or Motorola/IBM's PowerPCproduct line. The design is simple for someone familiar with the stateof the art given the above pseudocode and description. The storage unit410 includes RAM when using a digital processor.

In addition, a person familiar with the state of the art could implementthe process with analog and digital circuitry, either separate or in anapplication specific integrated circuit (ASIC). The analog and digitalcircuitry could include any combination of the following devices: adigital-to-analog converter (D/A), comparators, sample-and-holdcircuits, delay elements, analog-to-digital converter (A/D), andprogrammable logic controllers (PLC).

The foregoing descriptions of the preferred embodiments of the inventionhave been presented to teach those skilled in the art how to bestutilize the invention. Many modifications and variations are possible inlight of the above teaching. For example, even though this inventiondiscusses audio and the Internet, it is extendable to other types ofcontent and distribution. To this end, the following claims define thescope and spirit of the invention.

1. A method comprising: receiving audio or video content, the audio orvideo content comprising a first watermark associated therewith, thefirst watermark comprising first data indicating that the audio or videocontent is rights protected; and using at least a programmed electronicprocessor, associating a second watermark with the audio or videocontent, the second watermark comprising second data indicatingpermissible uses or restrictions associated with the audio or videocontent, wherein the second watermark is associated with the audio orvideo content later in time relative to the first watermark.
 2. Themethod of claim 1 wherein the first watermark is embedded in the audioor video content in a more robust manner, relative to embedding of thesecond watermark.
 3. The method of claim 1 wherein the second dataindicates whether the audio or video content can be copied orreproduced.
 4. The method of claim 1 wherein the second data associatesthe audio or video content with a particular user device.
 5. The methodof claim 1 wherein the second data associates the audio or video contentwith a particular user.
 6. The method of claim 1 wherein relative to thesecond watermark, the first watermark is more complex to detect.
 7. Themethod of claim 1 wherein the first watermark is embedded in the audioor video content through alterations to the audio or video content. 8.The method of claim 7 wherein the first watermark is generallyimperceptible to a human observer.
 9. The method of claim 7 wherein thesecond watermark is embedded in the audio or video content throughalterations to the audio or video content.
 10. The method of claim 9wherein the second watermark is generally imperceptible to a humanobserver.
 11. The method of claim 1 further comprising an act ofreproducing the audio or video content.
 12. The method of claim 11wherein said act of reproducing comprising distributing the audio orvideo content.
 13. The method of claim 1 wherein the first watermarkonly comprises two states: present, or absent.
 14. The method of claim 1wherein the second data comprises a link to a storage device.
 15. Themethod of claim 1 wherein the first watermark is out-of-band withrespect to the audio or video content.
 16. The method of claim 15wherein the second watermark is out-of-band with respect to the audio orvideo content.
 17. A method comprising: receiving audio or videocontent, the audio or video content comprising a first watermarkassociated therewith, the first watermark comprising first dataindicating that the audio or video content is rights protected, theaudio or video content further comprising a second watermark therewith,the second watermark comprising second data indicating permissible usesor restrictions associated with the audio or video content, wherein thesecond watermark is associated with the audio or video content later intime relative to the first watermark; and using a programmed electronicprocessor, associating a third watermark with the audio or videocontent, the third watermark comprising third data indicatingpermissible uses or restrictions associated with the audio or videocontent, wherein the third watermark is associated with the audio orvideo content later in time relative to the first watermark and thesecond watermark.
 18. The method of claim 17 wherein the first watermarkis embedded in the audio or video content in a more robust manner,relative to embedding of the second watermark and the third watermark.19. The method of claim 17 wherein at least one of the second data orthe third data indicates whether the audio or video content can becopied or reproduced.
 20. The method of claim 17 wherein relative to thesecond watermark and the third watermark, the first watermark is morecomplex to detect.
 21. The method of claim 17 wherein at least two ofthe first watermark, second watermark and third watermark are associatedwith the audio or video content by being embedded therein.
 22. Themethod of claim 17 wherein at least two of the first watermark, secondwatermark and third watermark are out-of-band with respect to the audioor video content.
 23. Them method of claim 17 further comprising an actof reproducing the audio or video content.
 24. The method of claim 23wherein said act of reproducing comprising distributing the audio orvideo content.
 25. The method of claim 17 wherein the first watermarkonly comprises two states: present, or absent.
 26. A method comprising:receiving audio or video content, the audio or video content comprisinga first watermark associated therewith, the first watermark comprisingfirst data indicating that the audio or video content is rightsprotected, the audio or video content further comprising a secondwatermark associated therewith, the second watermark comprising seconddata associated with a particular user or user device, wherein thesecond watermark is associated with the audio or video content later intime relative to the first watermark; detecting the second watermark toobtain the second data; and based at least in part on the second data,and using a programmed electronic processor, controlling reproduction ofthe audio or video content.
 27. The method of claim 26 wherein thesecond data comprises a link to a storage device.
 28. The method ofclaim 26 wherein the first watermark and the second watermark areembedded in the audio or video content through alterations to the audioor video content.
 29. The method of claim 28 wherein the alterations aregenerally imperceptible to a human observer of the audio or videocontent.
 30. The method of claim 26 further comprising: only if thesecond data is not obtained, detecting the first watermark to obtain thefirst data, wherein said act of controlling reproduction is then basedat least in part on the first data instead of the second data.
 31. Themethod of claim 30 wherein the first data only comprises two states:present, or absent.
 32. The method of claim 26 wherein the firstwatermark is embedded in the audio or video content in a more robustmanner, relative to embedding of the second watermark.
 33. The method ofclaim 26 wherein relative to the second watermark, the first watermarkis more complex to detect.
 34. A method comprising: obtaining audiocontent, the audio content including elements to be audibly rendered toa user, where embedded data is conveyed along with the audio content;analyzing the embedded data for rights data, wherein the rights data iscarried by a header; only if the rights data is missing, and using aprogrammed electronic processor, analyzing the embedded data for protectdata that is contained within the elements to be audibly rendered to auser; and if the rights data is missing and the protect data is present,then restricting processing of the audio content.
 35. The method ofclaim 34 wherein relative to detection of the protect data, the rightsdata is more efficient to detect.
 36. The method of claim 34 wherein theprotect data is generally imperceptible when the elements are audiblyrendered for the user.
 37. The method of claim 34 wherein relative tothe rights data, the protect data is more robust to removal with audioprocessing.
 38. The method of claim 37 wherein relative to the rightsdata, the protect data is more complex to detect.
 39. The method ofclaim 34 wherein the protect data is either present or absent.
 40. Themethod of claim 39 wherein the rights data includes copy control states,and the copy control states represent at least one of a plurality ofstates, the states comprising at least: i) copy once; ii) copy no more;and iii) copy never.
 41. The method of claim 40 wherein the copy controlstates further represent: iv) copy freely.
 42. The method of claim 34wherein the method is performed during a process of loading the audiocontent to a remote location.
 43. The method of claim 42 wherein theremote location comprises portable media, and the restricting processingof the audio content comprises blocking loading of the audio content tothe portable media.
 44. The method of claim 34 wherein the rights datacomprises multi-bits and carries different information than the protectdata.
 45. The method of claim 34 wherein the rights data and protectdata carry different information.
 46. The method of claim 34 wherein theprotect data is contained within the elements through steganographicembedding.
 47. A computer readable medium comprising instructions storedthereon, said instructions cause an electronic processor to perform themethod of claim
 1. 48. A computer readable medium comprisinginstructions stored thereon, said instructions cause an electronicprocessor to perform the method of claim
 17. 49. A computer readablemedium comprising instructions stored thereon, said instructions causean electronic processor to perform the method of claim
 26. 50. Acomputer readable medium comprising instructions stored thereon, saidinstructions cause an electronic processor to perform the method ofclaim
 34. 51. A programmed computing device comprising instructionsstored in memory, said instructions cause said programmed computingdevice to perform the method recited in claim
 1. 52. A programmedcomputing device comprising instructions stored in memory, saidinstructions cause said programmed computing device to perform themethod recited in claim
 17. 53. A programmed computing device comprisinginstructions stored in memory, said instructions cause said programmedcomputing device to perform the method recited in claim
 26. 54. Aprogrammed computing device comprising instructions stored in memory,said instructions cause said programmed computing device to perform themethod recited in claim
 34. 55. An apparatus comprising: electronicmemory comprising audio or video content, the audio or video contentcomprising a first watermark associated therewith, the first watermarkcomprising first data indicating that the audio or video content isrights protected; and an electronic processor programmed for associatinga second watermark with the audio or video content, the second watermarkcomprising second data indicating permissible uses or restrictionsassociated with the audio or video content, in which the secondwatermark is associated with the audio or video content later in timerelative to the first watermark.
 56. The apparatus of claim 55 in whichsaid electronic processor is operating to perform said associating asecond watermark with the audio or video content.
 57. An apparatuscomprising: electronic memory comprising received audio or video contentstored therein, the audio or video content comprising a first watermarkassociated therewith, the first watermark comprising first dataindicating that the audio or video content is rights protected, thereceived audio or video content further comprising a second watermarktherewith, the second watermark comprising second data indicatingpermissible uses or restrictions associated with the audio or videocontent, in which the second watermark is associated with the audio orvideo content later in time relative to the first watermark; and anelectronic processor programmed for associating a third watermark withthe received audio or video content, the third watermark comprisingthird data indicating permissible uses or restrictions associated withthe audio or video content, in which the third watermark is associatedwith the audio or video content later in time relative to the firstwatermark and the second watermark.
 58. The apparatus of claim 57 inwhich said electronic processor is operating to perform said associatinga third watermark with the received audio or video content.
 59. Anapparatus comprising: electronic memory for receiving audio or videocontent, the audio or video content comprising a first watermarkassociated therewith, the first watermark comprising first dataindicating that the audio or video content is rights protected, theaudio or video content further comprising a second watermark associatedtherewith, the second watermark comprising second data associated with aparticular user or user device, wherein the second watermark isassociated with the audio or video content later in time relative to thefirst watermark; detecting the second watermark to obtain the seconddata; and based at least in part on the second data an electronicprocessor programmed for controlling reproduction of the audio or videocontent.
 60. The apparatus of claim 59 in which said electronicprocessor is operating to perform said controlling reproduction of theaudio or video content.
 61. An apparatus comprising: electronic memoryfor buffering obtained audio content, the audio content includingelements to be audibly rendered to a user, where embedded data isconveyed along with the audio content; and an electronic processorprogrammed for: analyzing the embedded data for rights data, wherein therights data is carried by a header; only if the rights data is missing,analyzing the embedded data for protect data that is contained withinthe elements to be audibly rendered to a user; and if the rights data ismissing and the protect data is present, then restricting processing ofthe audio content.
 62. The apparatus of claim 61 in which saidelectronic processor is operating to perform at least one functionrecited therein.
 63. A method of managing processing of video content,the video content including picture data representing picture elementsto be rendered for display to a user, where embedded data is conveyedalong with the video content, the method including: (a) examining theembedded data for rights data, in which the rights data is carried by aheader; (b) only if the rights data is missing, and using a programmedelectronic processor, examining the embedded data for protect data thatis contained within the picture data; and (c) if the rights data ismissing and the protect data is present, then limiting processing of thevideo content.
 64. A computer readable medium comprising instructionsstored thereon, said instructions cause an electronic processor toperform the method of claim
 63. 65. A programmed computing devicecomprising instructions stored in memory, said instructions cause saidprogrammed computing device to perform the method recited in claim 63.